perf(rocm): head_dim=128 decode arm -- the ROCm half of #382, default OFF - #767

Implements .agents/specs/rocm-decode-attn-d128.md, which landed ahead of this change as #564.

The ROCm half of #382. The CUDA half merged as #425 (66399617); this mirrors it, adopting that arm's flag, default and stated reason rather than inventing new ones.

What changes

d == 128 — the Qwen3-dense / Llama / Mistral head size — reaches the fast decode kernels instead of falling through to the generic PagedAttnOnline. LoadRowEplBf16/StoreRowEplBf16 gain an EPL=4 (uint2) case beside the existing EPL=8/16; the dispatch gates and the two launch switches gain a d == 128 arm. No new kernel and no new algorithm — the kernel bodies were already generic over EPL.

Default OFF, opt in with VT_ATTN_DECODE_D128=1 — the same env var, default and reason as the merged CUDA arm. The arm is correctness-complete but not byte-exact against the kernel it replaces: warp-strided online softmax reduces the KV sequence in a different order than PagedAttnOnline's per-tile loop, so a greedy anchor can move at an exact bf16 tie. Shipping OFF keeps every existing golden byte-identical. The flip owes the near-tie razor, a distributional gate and regen under the ratified-tie rule, and per the spec must be argued per backend — see the reversal below. That is what keeps #382 open.

Reviewer note

Spec §4 item 3 writes the gate as (d == 128 && (decode_d128 || decode_wmma)). This commit implements it without the decode_wmma disjunct, which is what the same item's "Forward reference" paragraph instructs: VT_ATTN_DECODE_WMMA does not exist in the tree, and the flag lands with the rocWMMA arm on its own branch. The difference is intentional; it is visible in the diff before the note explaining it is.

Evidence

gfx1200 (RX 9060 XT, RDNA4, 32 CU), ROCm 7.2.3, $GPU_LOCK held. All figures are a same-binary flag A/B — no rebuild between arms — at 1024-token synthetic prompt, 128 generated, greedy, seed 0, 2 reps per cell agreeing within ~1%.

Qwen3-4B has no GQA fusion at any head_dim, so its 2.09x isolates the EPL widening from the fusion.

Qwen3.5-0.8B is the negative control and it earned its keep. Its head_dim is 256, so the d == 128 gate provably cannot reach it. Its first OFF rep came in a 33% outlier at 31.14 ms, which a blind 2-rep average would have reported as a ~1.2x "win" for a model the flag cannot affect. Re-run three times: 23.86 / 23.75 / 23.68 against ON's 23.52 / 23.57.

End-to-end output throughput rises less than TPOT on the same runs (0.6B 2.48x, 1.7B 2.05x, 4B 2.02x) because they carry a 1024-token prefill the flag does not touch. TPOT isolates decode; throughput dilutes it.

Concurrency — the advantage grows, it does not compress

Qwen3-1.7B, --num-prompts = 2x concurrency:

This refuted the prediction made before the run, which reasoned that a tiny grid at concurrency 1 flatters the fast kernel. The dominant effect is the reverse: from c8 to c16 the fallback scales only 1.33x against the arm's 1.54x, and scaling efficiency at c16 relative to perfect-linear-from-c1 is 38% OFF against 58% ON. PagedAttnOnline is the batch-scaling bottleneck, not merely slow per call, so the win is largest in the regime a server actually runs in.

The c1 row reproduces an independently-run four-model sweep to within ~1% (52.85/21.93 there vs 53.40/22.26 here).

Correctness

• ctest -R 'rocm|cross_device' 5/5, chained directly to the exact-SHA push.
• New case: "paged attention at Qwen3 geometry (bf16, GQA 2, head_dim 128) matches the CPU oracle", looped over RegisteredDevices(), NMSE <= 5e-4 vs the CPU oracle plus OpProviderStats::declines == 0. Genuinely new coverage — the existing generic cross-device test runs at d=8, f32 and never reached any bf16 EPL-templated kernel, so none of them had bf16 coverage in this suite. (The merged CUDA arm shipped with no test at all.)
• Because the arm ships OFF and its flag is read into a static const bool — once per process — the default registration can only ever gate the fallback. tests/CMakeLists.txt adds a second invocation with the flag set, same shape as the existing test_dense_gateup_fused_marlin_off_* pair. Verified non-vacuous against the test_voxtral_e2e reports SUCCESS with zero assertions when its weights env var is unset (and ~40 more gates share the shape) #463 trap: 1 case, 6 assertions, not zero.
• Full ctest 448/455. The 7 failures are proven pre-existing, not asserted: a clean main 2784dd7b worktree built from source with none of this code fails the identical set (only test_op_parity's index shifts 403 -> 404, from the added registration). They are a missing shellcheck, an mmap-RSS assertion, a JSON type error, and the SharedExpertGate ROCm registration gap owed to unmerged feat(rocm): MoE combine/gate ops — SharedExpertGate, MoeCombine, MoeCombineGate (issue #41) #509.
• agent-preflight fails 9, a strict subset of that same baseline's 10 (differing only by role-undeclared). check-commit-trailers and check-doc-checkpoint both pass against this base.

Carried finding

#382 measured this same EPL=4 arm 1.6x slower on sm_110 / Jetson AGX Thor, where gfx1200 measures it 2-3.6x faster. Recorded, not reconciled — different kernels, different fallbacks, different memory systems. It is why the default-ON flip must be argued per backend rather than once, and it is preserved in the spec rather than averaged away.

Against the pinned oracle: 6.35x to 1.75x slower on per-token decode

Measured after the tables above, with both sides in the same container, oracle = vLLM 555967922 in its production configuration via vllm bench serve. Qwen3-0.6B, 1024 in / 128 out, concurrency 1, 8 prompts, warmup discarded, 3 reps:

Running our binary against the container's ROCm rather than the host's is a substitution, so it was proved inert first: in-container matches native at 42.79 vs 42.53 ms unset, and 12.03 vs 11.78 ms with the flag.

The prompt count is load-bearing. At --num-prompts 2 the oracle returned TPOT 6.96 ms and 13.45 ms on consecutive reps, a ~2x spread averaging to a plausible-looking and entirely fictional number. At 8 prompts with a discarded warmup both sides hold to ~±0.3 ms.

This number lived only in a PR comment, which a squash merge does not carry into the tree. It is now in the spec's §5 and appended to .agents/benchmark-record.md, with its caveats attached rather than trailing.

The container/glibc blocker was RETRACTED

An earlier revision of this body, and the spec's §6, said the oracle re-measure was blocked on a Nix-glibc vs container-glibc ABI mismatch. That diagnosis was wrong and is retracted. Our binary runs inside the pinned oracle container; the earlier failures were self-inflicted (LD_LIBRARY_PATH exported container-wide, which breaks the container's own tools, plus a bind mount that silently yielded nothing and looked exactly like a missing ELF interpreter). §6 now reads "not run — not blocked", and the WMMA-spec cross-reference is gone. A false blocker in the record is worse than no record, because it stops the next person from trying.

Not claimed

This does not close #488. That issue asks for a per-call kernel comparison and explicitly asserts no cause. The number above is per-token latency with asymmetric harnesses — the oracle over HTTP via vllm bench serve, ours in-process — so TPOT is the only comparable axis, and TTFT, E2EL and end-to-end throughput carry the oracle's HTTP and tokenizer overhead and are directional only. It is not the same-tool per-call trace AGENTS.md wants before a throughput claim. rocprofv3 is present in the container and our binary traces under it; what is still owed is decode-phase windowing on the oracle side, or the trace compares our decode against vLLM's model load and graph capture. One board, one shape. docs/BENCHMARKS.md's ROCm axis stays PENDING, and the row this PR adds is marked DIRECTIONAL and sits beside the existing row rather than overwriting it.

The flag-ON arm still has no proof it REACHES the new kernel, now filed as #1134. RegisteredDevices() (tests/vt/test_backend_cross_device.cpp:84-96) enumerates {kCUDA, kMETAL, kVULKAN, kXPU, kROCM} and excludes kCPU, so on a CPU-only runner — which is what CI has — the new case reports 1 test case, 0 assertions, exit 0, for both registrations. On ROCm the case's only backend assertion is OpProviderStats::declines == 0, counted at provider granularity, so it is identical with the flag set and unset. §9 stop condition 2 is left OPEN. The spec disclosed this honestly; what was missing is the issue AGENTS.md requires for a known gap not fixed in flow. Searched before filing: not a duplicate of #463 (the unset-weights-env-var shape, which does not describe the declines half), #785 (a kernel that never LAUNCHES behind a dead #if) or #900 (same family, LTX-2.5 subject).

Also out of scope and named in the spec's new ## Owed section: the dtype gap (ROCm's decode-opt is bf16-only at every head_dim, so 4 of 5 dtype combinations still fall to PagedAttnOnline at d=128 — pre-existing, inherited, not introduced), qg=4/qg=8 fusion, d=128 prefill, and the rocWMMA arm.

Record repairs carried in the final commit

docs(BACKEND-ROCM): retract the blocker, keep the oracle number, and file the gap, on top of joral's commits, which are untouched. It carries the retraction above; the oracle number into §5 and .agents/benchmark-record.md; docs/BENCHMARKS.md and docs/STATUS.md reconciled; §7's stale "two flag-on ctest registrations" corrected to one, matching what 8aedd780 already fixed in §4 item 3 and the Test-coverage section; a literal ## Owed heading over the owed list; and #1134 filed and appended to .agents/issue-index.md. Two comment-only edits at rocm_paged_attn.hip:330 and :455, which still enumerated "EPL=8 → d=256, EPL=16 → d=512" without the new EPL=4 case although the top-of-file comment at :264 had been updated.

The branch is rebased onto origin/main d1e5e9bc — it was 39 commits behind, and the rebase drops the earlier merge: upstream/main commit. Gates rerun from the worktree with explicit SHAs: check-commit-trailers, check-commit-style, check-doc-checkpoint, check-public-doc-tables, check-agent-record and check-pr-size, all OK. Not rebuilt and not re-run on hardware: every gfx1200 figure here is joral's, unchanged.

Row: BACKEND-ROCM
Issue: #382
Issue: #1134
Spec: #564

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]